EnVE: A consistent hybrid ensemble/variational estimation strategy for multiscale uncertain systems

Chaotic systems are characterized by long-term unpredictability. Existing methods designed to estimate and forecast such systems, such as Extended Kalman filtering (a “sequential” or “incremental” matrix-based approach) and 4DVar (a “variational” or “batch” vector-based approach), are essentially based on the assumption that Gaussian uncertainty in the initial state, state disturbances, and measurement noise lead to uncertainty of the state estimate at later times that is well described by a Gaussian model. This assumption is not valid in chaotic systems with appreciable uncertainties. A new method is thus proposed that combines the speed and LQG optimality of a sequential-based method, the non-Gaussian uncertainty propagation of an ensemble-based method, and the favorable smoothing properties of a variational-based method. This new approach, referred to as Ensemble Variational Estimation (EnVE), is a natural extension of the Ensemble Kalman and 4DVar algorithms. EnVE is a hybrid method leveraging sequential preconditioning of the batch optimization steps, simultaneous backward-in-time marches of the system and its adjoint (eliminating the checkpointing normally required by 4DVar), a receding-horizon optimization framework, and adaptation of the optimization horizon based on the estimate uncertainty at each iteration. If the system is linear, EnVE is consistent with the well-known Kalman filter, with all of its well-established optimality properties. The strength of EnVE is its remarkable effectiveness in highly uncertain nonlinear systems, in which EnVE consistently uses and revisits the information contained in recent observations with batch (that is, variational) optimization steps, while consistently propagating the uncertainty of the resulting estimate forward in time.